Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(aws-dynamodb-global): support for global dynamodb tables #2082

Closed
wants to merge 1 commit into from
Closed

feat(aws-dynamodb-global): support for global dynamodb tables #2082

wants to merge 1 commit into from

Conversation

KingOfPoptart
Copy link
Contributor

@KingOfPoptart KingOfPoptart commented Mar 22, 2019


Pull Request Checklist

  • Testing
    • Unit test added (prefer not to modify an existing test, otherwise, it's probably a breaking change)
    • CLI change?: coordinate update of integration tests with team
    • cdk-init template change?: coordinated update of integration tests with team
  • Docs
    • jsdocs: All public APIs documented
    • README: README and/or documentation topic updated
  • Title and Description
    • Change type: title prefixed with fix, feat will appear in changelog
    • Title: use lower-case and doesn't end with a period
    • Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
    • Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"
  • Sensitive Modules (requires 2 PR approvers)
    • IAM Policy Document (in @aws-cdk/aws-iam)
    • EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
    • Grant APIs (only if not based on official documentation with a reference)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

@KingOfPoptart KingOfPoptart requested a review from a team as a code owner March 22, 2019 18:19

Here is a minimal deployable Global DynamoDB tables definition:

```javascript
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typescript

super(scope, id, props);
const codeLocation = path.resolve(__dirname, "lambda", "handler.js.lambda");
this.lambdaFunctionContent = fs.readFileSync(codeLocation, "utf8");
this.lambdaFunction = new lambda.SingletonFunction(this, id + "-SingletonLambda", {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why append to the id, you could just use SingletonLambda as the id.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found it easier to track the deployed stacks in CFN by having a common prefix in the stack ID. Otherwise, its difficult to keep track

constructor(scope: cdk.Construct, id: string, props: DynamoDBGlobalStackProps) {
super(scope, id, props);
const codeLocation = path.resolve(__dirname, "lambda", "handler.js.lambda");
this.lambdaFunctionContent = fs.readFileSync(codeLocation, "utf8");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use lambda.Code.asset(codeLocation) instead of reading yourself.

.addAction("application-autoscaling:DeregisterScalableTarget")
.addAction("dynamodb:CreateGlobalTable")
.addAction("dynamodb:DescribeLimits")
.addAction("dynamodb:UpdateGlobalTable"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dynamodb:* permissions should be offered as static methods, either in this package or @aws-cdk/aws-dynamodb.

.addAction("dynamodb:CreateGlobalTable")
.addAction("dynamodb:DescribeLimits")
.addAction("dynamodb:UpdateGlobalTable"));
this.customResource = new cfn.CustomResource(this, id + "-CfnCustomResource", {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, why append to the id?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above

@@ -0,0 +1,81 @@
{
"name": "@aws-cdk/aws-dynamodb-global",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need a new package as opposed to @aws-cdk/aws-dynamodb?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made more sense to me to have a new package since it has different requirements than the regular aws-dynamodb table - i.e. required tableName and streamSpecification. As well as its a higher level object than the dynamodb package.

Once CFN officially supports global-dynamodb tables, it would probably be easier to deprecate this package than remove it from an integrated portion of the aws-dynamodb package

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to have this as a separate module. Maybe we can add a comment in the dynamodb README that mentions it

import globaldynamodb = require('@aws-cdk/aws-dynamodb-global');
const CONSTRUCT_NAME = 'aws-cdk-dynamodb-global';
const TABLE_PARTITION_KEY: Attribute = { name: 'hashKey', type: AttributeType.String };
const STACK_PROPS: DynamoDBGlobalStackProps = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not idiomatic, pass the properties directly when creating the construct:

new GlobalTable(stack, 'aws-cdk-dynamodb-global', {
    dynamoProps: {
        partitionKey: TABLE_PARTITION_KEY,
        tableName: TABLE_NAME
    },
    regions: [ 'us-east-1', 'us-east-2', 'us-west-2' ],
    // etc.
});

```

## Implementation Notes
Global Dynamodb tables is an odd case currently. The way this package works -
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there documentation on the recommended practice from AWS?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AWS has no recommended practice for this since its not supported in CFN currently. This is the solution determined by myself and by chatting with CDK contributors

@KingOfPoptart
Copy link
Contributor Author

Changes have been posted to the PR, and responses to questions posted.

Copy link
Contributor

@eladb eladb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just reviewed the readme so far


```typescript
import globaldynamodb = require('@aws-cdk/aws-dynamodb-global');
const CONSTRUCT_NAME = 'aws-cdk-dynamodb-global';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These consts are not really adding much in terms of readability and they are only used once, just inline them

packages/@aws-cdk/aws-dynamodb-global/README.md Outdated Show resolved Hide resolved
```

## Implementation Notes
Global Dynamodb tables is an odd case currently. The way this package works -
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure you capitalize service names properly and use the prefix "Amazon" or "AWS" (in this case it's Amazon DynamoDB).


### Notes

`DynamoDBGlobalStackProps.dynamoProps` is essentially a copy of `dynamodb.TableProps` - however `tableName` is *required*.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can extract all the props besides tableName from TableProps into TableOptions interface and then have GlobaProps and TableProps inherit from Options (that's kind of how we do this)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// in @aws-cdk/aws-dynamodb

export interface TableOptions {
  // all props besides `tableName`
}

export interface TableProps extends TableOptions {
  tableName?: string;
}

// in aws-dynamodb-global
export interface GlobalStackProps extends TableOptions {
  tableName: string;
}


`DynamoDBGlobalStackProps.dynamoProps` is essentially a copy of `dynamodb.TableProps` - however `tableName` is *required*.

GlobalTable() will override any set `dynamoProps.streamSpecification` to be `NEW_AND_OLD_IMAGES` since this is a required attribute for global dynamodb tables to work.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably should error if users specified something else

@KingOfPoptart
Copy link
Contributor Author

More changes pushed.

@KingOfPoptart
Copy link
Contributor Author

bump

* Properties for the mutliple DynamoDB tables to mash together into a
* global table
*/
export interface DynamoDBGlobalStackProps extends cdk.StackProps {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to GlobalTableProps

* Properties for the mutliple DynamoDB tables to mash together into a
* global table
*/
export interface GlobalTableProps extends cdk.StackProps {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I probably didn't explain myself too well (interfaces can extend multiple bases):

export interface GlobalTableProps extends cdk.StackProps, dynamodb.TableOptions {
  tableName: string;
  regions: string[];
}

Then, you don't need GlobalDynamoDBProps at all.

We like flat ;-)

regions: string[];
}

export class GlobalTable extends cdk.Construct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Short description of this type (the README intro is great)

@@ -0,0 +1,3 @@
export * from "./aws-dynamodb-global";
export * from "./lambda-global-dynamodb";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to export this module? Seems like an internal implementation

@@ -0,0 +1,81 @@
{
"name": "@aws-cdk/aws-dynamodb-global",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to have this as a separate module. Maybe we can add a comment in the dynamodb README that mentions it

const STACK_NAME = 'global-dynamodb-integ';

// DynamoDB table parameters
const TABLE = 'GlobalTable';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these consts needed?

/**
* Creates dynamoDB tables across regions that will be able to be globbed together into a global table
*/
public tables: MultiDynamoDBStack[];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the API should only expose the Table objects and not the stack that wraps them. Furthermore, I don't even thing you need this MultiDynamoDBStack class (and of course you don't need it in the public API). You can just create those stacks in-place:

for (const reg of props.regions) {
  const regionalStack = new cdk.Stack(this, '...', { env: { region: reg } });
  const regionalTable = new dynamodb.Table(regionalStack, 'Table', props);
  this.tables.push(regionalTable);
}

This is how you should define the property:

public readonly tables = new Array<dynamodb.Table>();

P.S. this is not ideal because the array is mutable. The "right" thing to do would be to return a clone:

private readonly _regionalTables = new Array<dynamodb.Table>();

public get regionalTables() {
  return this._regionalTables.map(x => x);
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chiming in—if mutability is a concern, would it make sense to use a library like https://github.com/immutable-js/immutable-js, or would that be too big of a dependency to expose as a public API in the CDK?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We prefer to avoid taking such dependencies (and frankly JS/TS are sufficient). Soon Typescript will have x: readonly string[] and the world will be whole.

export class GlobalTable extends cdk.Construct {

/**
* Creates dynamoDB tables across regions that will be able to be globbed together into a global table
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

call this regionalTables

},
});
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't check in the handler.zip file! Where is the code and how is it bundled into a zip file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's been addressed now

@RomainMuller RomainMuller added feature-request A feature should be added or improved. @aws-cdk/aws-dynamodb Related to Amazon DynamoDB labels Mar 28, 2019
* global table
*/
export interface RegionalTablesProps extends cdk.StackProps, dynamodb.TableOptions {
/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 spaces please

*/
export interface RegionalTablesProps extends cdk.StackProps, dynamodb.TableOptions {
/**
* Enforces a particular table name to share across Global DynamoDB tables
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wording here feels off. "Enforces?" Ideally if we can copy & paste from official AWS docs that would be better. I would write something like "Name of the DynamoDB table to use across all regional tables. This is required for global tables".

}

/**
* Global Tables builds upon DynamoDB’s global footprint to provide you with a fully managed, multi-region,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This describes Global Tables but the class is RegionalTables?

* Properties for the mutliple DynamoDB tables to mash together into a
* global table
*/
export interface RegionalTablesProps extends cdk.StackProps, dynamodb.TableOptions {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this called "RegionalTables" instead of "GlobalTable"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You asked for the class to be renamed?
#2082 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not the class, the property name. It's: GlobalTable.regionalTables. Otherwise it doesn't make sense...

constructor(scope: cdk.Construct, id: string, props: RegionalTablesProps) {
super(scope, id);
this._regionalTables = [];
if ( props.streamSpecification != null && props.streamSpecification !== dynamodb.StreamViewType.NewAndOldImages) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Remove space after "("

} else if (event.RequestType === "Update") {
console.log("UDPATE!");
// Put your custom update logic here
sendResponse(event, context, "SUCCESS", { Message: "Resource update successful!" });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it makes sense that update is a no-op? What if I add a region for example?

} else {
console.log("FAILED!");
sendResponse(event, context, "FAILED");
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You must catch all exceptions and send a FAILED response. Otherwise, any exception will cause the CFN deployment to timeout after 4 hours.

Then, all failure cases should just "throw".

Ideally, if you can also have only a single point in your code that sends a SUCCESS response, it would be safer, especially if someone updates this code in the future and forgets to handle some case (alternatively, "throw" when you fall through).

}
};

function setupWatchdogTimer(event, context, callback) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

context.done();
});

// write data to request body
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a unit test for this handler

@@ -0,0 +1,3 @@
'use strict';

const handler = require('../lib/lambda/handler');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not enough.. You'll have to mock stuff..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol I know, just checked in what I had gotten done so far this morning :)

@eladb
Copy link
Contributor

eladb commented Apr 11, 2019

@KingOfPoptart could you please rebase this and we'll merge that in

@KingOfPoptart
Copy link
Contributor Author

This PR has been replaced by #2251

@KingOfPoptart KingOfPoptart deleted the KingOfPoptart/globaldynamodb branch April 16, 2019 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-dynamodb Related to Amazon DynamoDB feature-request A feature should be added or improved.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants